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REMARKS 

This is a Response to the Office Action mailed August 25, 2009, in which a three 
(3) month Shortened Statutory Period for Response has been set, due to expire November 25, 
2009. Thirty-two (32) claims, including four (4) independent claims, were paid for in the 
application. No new matter has been added to the application. No fee for additional claims is 
due by way of this Amendment. The Director is authorized to charge any additional fees due by 
way of this Amendment, or credit any overpayment, to our Deposit Account No. 19-1090. 
Claims 33-47 and 49-61 are pending. 

Information Disclosure Statement 

Applicants filed a Supplemental IDS on June 5, 2009. It is respectfully requested 
that an initialed copy of the IDS be provided to Applicants to confirm consideration thereof 

35 U.S.C. S102fb^ Rejections 

Claims 33-47 and 49-61 were rejected under 35 U.S.C. § 102(b) as being 
anticipated by Multisensory Visual Servoing by a Neural Network by Wei et al. (hereinafter 
"Wei"). 

Claim 50 is independent so will be discussed before the claims that depend 
therefrom. Claim 50 recites: 

A method usefiil in three-dimensional pose estimation for use with a single 
camera mounted to a movable portion of a robot, the method comprising: 
capturing a two-dimensional image of a volume containing a target object; 
locating a number of features in the captured image of the target object; and 
determining an object space-to-camera space transformation for the target 
object based at least in part on a position of at least some of the located 
features using only the single captured image and an algorithm that employs a 
known or determinable physical relationship between at least some of the 
located features. 
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The Office alleges that Wei teaches all of the elements recited by claim 50, 
relying on Figures 2-3 of Wei. However, Figure 2 refers to cameras in the plural. Figure 3 
shows two cameras. See Wei, at page 4 bottom of first column-top of second column, Figure 3, 
and page 1, second column. Figure 3 is reproduced below, in which circles have been added to 
identify the two stereo cameras. 




The Office also relies on Figure 4 of Wei. However Figure 4 illustrates stereo 
images produced from the two cameras. 

It is clear that Wei does not teach or suggest "[a] method useful in three- 
dimensional pose estimation for use with a single camera mounted to a movable portion of a 
robot, as recited by claim 50. (Emphasis added.) It is also clear that Wei does not teach or 
suggest "determining an object space-to-camera space transformation for the target object based 
at least in part on a position of at least some of the located features using only the single captured 
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image and an algorithm that employs a known or determinable physical relationship between at 
least some of the located features." (Emphasis added). 

Claim 33 is dependent from claim 50 and fiirther recites "capturing a number of 
images of a calibration object by the camera; determining a set of intrinsic parameters of the 
camera from at least one of the number of images of the calibration object captured by the 
camera; and determining a set of extrinsic parameters of the camera from at least one of the 
number of images of the calibration object captured by the camera, the set of extrinsic 
parameters comprising a camera space-to-training space transformation defining a 
transformation between a camera space reference frame and a training space reference frame." 
The Office alleges that such is taught by Wei, relying on page 1, column 1. However, the cited 
portion of Wei actually appears to teach away from such (e.g. "On the contrary, a neural network 
approach for doing the same job can avoid all such calibrations and provides convenient sensor 
fusion framework. The network learns the direct transformation from (multi-) sensory data to 
the required motions based on a set of examples. The effects of sensor calibration, geometric 
transformation, and the relative importance of each sensor are all absorbed in a single 
network."). 

Claim 35 is indirectly dependent from claim 50 and further recites "positioning 
the camera orthogonally with respect to a ruled template with a number of features, where a 
known or determinable physical relationship exists between at least some of the features." The 
Office contends that Wei teaches such, relying in particular on table II, and by interpreting table 
II to be a template. Applicants respectfully contend that a table in a printed publication cannot 
be reasonably interpreted to constitute a ruled template with a number of features. In any case, 
Wei clearly does not teach or suggest positioning a camera orthogonally with respect to such 
table. 

Claim 37 is indirectly dependent from claim 50 and fiirther recites "capturing a 
number of images of a calibration object by the camera comprises capturing at least one image at 
each of a plurality of positions spaced perpendicularly from the calibration object." The Office 
alleges that Wei teaches such, replying on Figures 2-4. Figure 2 illustrates the overall system, 
while Figure 3 shows an end effecter with two cameras in a stereoscopic arrangement and 
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number of laser range finders. Figure 4 illustrates stereo images as captured by the two cameras. 
(Note, since the two cameras are not parallel, than it is impossible for both cameras to be 
perpendicular to an object at a same time. ) The positions of the cameras do not appear to be 
perpendicular with respect to the object. In any case, Wei clearly does not teach capturing a 
number of images at a plurality of positions spaced perpendicularly from the calibration object" 
(emphasis added) as recited by claim 37. 

Claim 39 is indirectly dependent from claim 50 and further recites "determining at 
least one of a focal length, a first order radial lens distortion coefficient, a set of coordinates of a 
center of a radial lens distortion, or a scale factor indicative of a fi-amegrabber scanline 
resampling uncertainty." The Office alleges that such is taught by Wei, but does not identify any 
specific portion of Wei. Upon review of Wei, Applicant's attorney has not found any teaching of 
such in the cited reference. Applicants respectfully ask that the Office specifically point out such 
teaching, if the Office persists in this rejection. 

Claim 40 is indirectly dependent fi-om claim 50 and further recites that 
"determining a set of extrinsic parameters of the camera fi-om at least one of the number of 
images of the calibration object captured by the camera, the set of extrinsic parameters 
comprising a camera space-to-training space transformation defining a transformation between a 
camera space reference frame and a training space reference frame comprises determining a 
respective translation component along three orthogonal axes, and a respective rotation 
component about the three orthogonal axes." The Office alleges that such is taught by Wei, 
relying on page 1, col. 1, second paragraph, and figures 2-3. However, the cited portion of Wei 
actually appears to teach away from such (e.g. "On the contrary, a neural network approach for 
doing the same job can avoid all such calibrations and provides convenient sensor fusion 
fi-amework. The network learns the direct transformation fi-om (multi-) sensory data to the 
required motions based on a set of examples. The effects of sensor calibration, geometric 
transformation, and the relative importance of each sensor are all absorbed in a single 
network."). 

Claim 41 is indirectly dependent from claim 50 and further recites "determining a 
camera space-to-tool space transformation based at least in part on at least two of the number of 
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images captured by the camera of the calibration object." The Office alleges Wei teaches such, 
relying on Figures 2-4. Figure 2 illustrates the overall system. Figure 3 shows an end effecter 
carrying a pair of cameras and laser range finders, and Figure 4 shows a stereo images captured 
by the pair of cameras. Applicant does not see how the Office could possible infer such details 
of operation {i.e., determining a camera space-to-tool space transformation based at least in part 
on at least two of the number of images captured by the camera of the calibration object ) from 
such high level Figures. 

Claim 42 is indirectly dependent fi-om claim 50 and further recites "determining a 
camera space-to-tool space transformation based on single one of the number of images captured 
by the camera of the calibration object and on a number of physical coordinates of at least one 
feature of the calibration object." The Office alleges Wei teaches such, relying on Figures 2-4. 
Figure 2 illustrates the overall system, Figure 3 shows an end effecter carrying a pair of cameras 
and laser range finders, and Figure 4 illustrates stereo images captured by the pair of cameras. 
Such cannot possibly teach or suggest , "determining a camera space-to-tool space 
transformation based on single one of the number of images captured by the camera of the 
calibration object and on a number of physical coordinates of at least one feature of the 
calibration object" as recited in claim 42 (emphasis added). Wei simply does not teach or 
suggest a single camera single image approach to robotic machine-revision. 

Claim 43 is dependent from claim 50 and further recites, inter alia, "selecting a 
nimiber of features from the captured image of the teaching object; and determining a set of 
object space coordinates for each of the selected features from the captured image of the teaching 
object." The Office alleges Wei teaches such, relying on page 2, col. 1 and Figure 4. However, 
the cited portion does not appear to refer to features, let alone "selecting features and 
determining object space coordinates for each of the selected features" as recited in claim 43. 
The cited portion is directed to training and with dealing with inexactness of mapping through a 
recursive method. 

Claim 44 is indirectly dependent from claim 50 and further recites "selecting a 
number of features from the captured image of the teaching object comprises selecting six 
features from the captured image of the teaching object." The Office alleges Wei teaches such. 
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again relying on page 2, col. 1 and Figure 4. However, the cited portion does not appear to refer 
to features, let alone "selecting six features from the captured image of the teaching object" as 

recited in claim 44. 

Claim 45 is indirectly dependent from claim 50 and fiirther recites "determining 
an object space-to-camera space transformation defining a transformation between an object 
space reference frame and the camera space reference frame." The Office alleges that Wei 
teaches such relying on page 2, col. 1, and Figures 2-4, "wherein reference frame has been 
considered as position frame." It appears that the Office is employing the "position frame" to be 
both the object space reference frame and the camera space reference frame. Such is of course 
incorrect since two different reference frames have been recited, and the Office cites to at most a 
single reference frame in its rejection. 

Claim 46 is indirectly dependent from claim 50 and further recites "determining a 
position and an orientation of an object frame in the tool frame reference frame based at least in 
part on the object frame-to-camera space and camera space-to-tool space transformations." The 
Office alleges that Wei teaches such relying on pages 1-22, the abstract, and Figures 2-4. While 
Wei mentions a transformation, the cited passages do not appear to describe the various 
transformations or various reference frames recited in claim 46. 

Claim 47 is indirectly dependent from claim 50 and further recites "providing the 
position and orientation of the object frame to the robot; and training an intended operation path 
inside the object frame." The Office alleges that Wei teaches such relying on Figure 4 and the 
abstract. As previously explained. Figure 4 illustrates stereoscopic images captured by a pair of 
cameras. The abstract discuss camera calibration and sensor to hand calibration {e.g., camera to 
end effecter). The abstract further discusses supplying sensor data to a multilayer feed forward 
neural network to associate the direct transformation for the sensory data to required motions, 
which avoids the need for network retaining. There is no discussion of providing position and 
orientation of an object frame to the robot or of training an intended operation path inside the 
object frame, as opposed to any other possible reference frame. In fact, it appear that only small 
portions of motions M are provided to the robot, and without more, one can only assume that 
those are in a robot frame of reference. 
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Claim 49 is dependent from claim 50 and ftirther recites "adjusting a position of 
the movable portion of the robot if the number of features located in the captured image of the 
target object is determined to be an insufficient number of features; and capturing another two- 
dimensional image of the volume containing the target object before determining the object 
space-to-camera space transformation for the target object." The Office alleges that Wei teaches 
such relying on page 2, col. 1, and Figure 2-4. Applicant's attorney has carefully reviewed the 
cited passage, and cannot find any reference to features or to the number of located features, let 
alone adjusting a position in response to the number of features being insufficient or capturing an 
additional image in response to the same. 

Claim 51 is dependent from claim 50 and further recites "determining at least one 
movement of the robot that orients the camera orthogonally with respect to the target object 
based at least on part on the object space-to-camera space transformation." The Office alleges 
that such is taught by Figure 4 of Wei. However, Figure 4 illustrates stereo images which appear 
to have a random orientation with respect to the workpiece. There is nothing in Figure 4 that 
would suggest an orthogonal orientation between the camera and the workpiece. 

Claim 52 is dependent from claim 50 and ftirther recites "determining a position 
of the object frame in the tool space reference frame; and providing an object frame to the 
robot." The Office alleges that such is taught by Wei, relying on page 2, col. 1, and Figures 3 
and 4. While Figure shows an image of an object, such is not the same as the recited "object 
frame" not the recited "providing an object frame to the robot." Likewise, while Figure 3 shows 
an end effecter portion of a robot, such is not the same as the recited "providing an object frame 
to the robot." In column 1 of page 2, Wei discusses training a neural network and dealing with 
inexactness of mapping. According to the cited passage, only a small portion of Mi is sent to the 
robot, where Mi is an amount of motion of the robot hand with respect to the reference position 
and the sensory data which was recorded after each motion. Wei, page 2, col. 1 . Thus, there is 
no basis to allege that Wei teaches "determining a position of the object frame in the tool space 
reference frame; and providing an object frame to the robot" as recited by claim 52. 

Claim 53 is independent and recites, inter alia, "a single camera operable to 
capture a number of images of a calibration object" "determining an object space-to-camera 
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space transformation based at least in part on a position of at least some of the located features in 
solely the captured image using an algorithm that employs a known or determinable physical 
relationship between at least some of the located features." The Office alleges that Wei teaches 
such. As previously explained, Wei is clearly teaches the use of two cameras to produce stereo 
images. Thus, Wei is inapposite with respect to the single camera approach recited in claim 53. 

Claim 54 is dependent from claim 53 and further recites, inter alia, "determining 
a set of object space coordinates for each of the selected features from the captured image of the 
teaching object" and "determining an object space-to-camera space transformation defining a 
transformation between an object space reference frame and the camera space reference frame." 
The cited passage of Wei {i.e., page 2, col. 1) does not each such. The cited passage at most 
teaches sending only a small portion of Mi to the robot, where Mi is an amount of motion of the 
robot hand with respect to the reference position and the sensory data which was recorded after 
each motion. Wei, page 2, col. 1 . 

Claim 57 is dependent from claim 53 and fiirther recites "adjusting a position of 
the movable portion of the robot if the number of features located in the captured image of the 
target object is determined to be an insufficient number of features." The Office alleges such is 
taught is page 2, col. 1 of Wei. Applicant's attorney has carefully reviewed the cited passage, 
and cannot find any reference to features or to the number of located features, let alone adjusting 
a position in response to the number of features being insuflficient. 

Claim 58 is independent recites, inter alia, "a single camera operable to capture a 
number of images of a calibration object;" and "determining an object space-to-camera space 
transformation based at least in part on a position of at least some of the located features using 
the captured image without any additional captured images and an algorithm that employs a 
known or determinable physical relationship between at least some of the located features." 

As previously explained, Wei is clearly teaches the use of two cameras to produce 
stereo images. Thus, Wei is inapposite with respect to the single camera approach recited in 
claim 58. 

Claim 61 is dependent from claim 58 and fiirther recites "adjusting a position of 
the movable portion of the robot if the number of features located in the captured image of the 
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target object is determined to be an insufficient number of features." The Office alleges such is 
taught is page 2, col. 1 of Wei. Applicant's attorney has carefully reviewed the cited passage, 
and cannot find any reference to features or to the number of located features, let alone adjusting 
a position in response to the number of features being insufficient. 

Conclusion 

Applicants respectfully submit that the pending claims are in condition for 
allowance. Any remarks in support of patentability of one claim should not be imputed to any 
other claim, even if similar terminology is used. Any remarks referring to only a portion of a 
claim should not be understood to base patentability on that portion; rather, patentability must 
rest on each claim taken as a whole. A number of clarifying amendments have also been made 
to the above claim set. Applicants do not acquiesce to each of the Examiner's rejections and to 
each of the Examiner's assertions regarding what the cited references show or teach, even if not 
expressly discussed herein. Although changes to the claims have been made, no acquiescence or 
estoppel is or should be implied thereby; such amendments are made only to expedite 
prosecution of the present application and are without prejudice to the presentation or assertion, 
in the fixture, of claims relating to the same or similar subject matter. 
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If the undersigned attorney has overlooked a relevant teaching in any of the 
references, the Examiner is requested to point out specifically where such teaching may be 
found. In light of the above amendments and remarks. Applicants respectfiilly submit that all 
pending claims are allowable. Applicants, therefore, respectfully request that the Examiner 
reconsider this application and timely allow all pending claims. The Examiner is encouraged to 
contact the undersigned by telephone to discuss the above and any other distinctions between the 
claims and the applied references, if desired. If the Examiner notes any informalities in the 
claims, the Examiner is encouraged to contact the undersigned by telephone to expediently 
correct such informalities. 

Respectfully submitted, 

SEED Intellectual Property Law Group pllc 

/Frank Abramonte/ 

Frank Abramonte 
Registration No. 38,066 

FXA:sc 

701 Fifth Avenue, Suite 5400 
Seattle, Washington 98104 
Phone: (206)622-4900 
Fax: (206)682-6031 
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